Computing machine, learning method of classifier, and analysis system

ABSTRACT

A computing machine includes: a storage unit that stores learning data; a learning unit that executes learning processing for generating a classifier by using the learning data; and a generation unit that generates the learning data, and the generation unit calculates a feature amount vector handled by the classifier by using the learning data stored in the storage unit, analyzes the distribution ox the learning data in a feature amount space on the basis of the feature amount vector to specify a boundary where the classification result of the classifier changes in the feature amount space, and generates new learning data by using the learning data existing in the vicinity of the boundary.

INCORPORATION BY REFERENCE

This application claims the priority of Japanese Patent Application No.2020-18575, filed on Feb. 6, 2020, and the content thereof isincorporated herein by reference.

TECHNICAL FIELD

The present invention relates to a generation technique of a classifierfor classifying an arbitrary event.

BACKGROUND ART

In order to analyze the components or the like of a sample such as bloodand urine, an automatic analysis system including an immunoanalysisdevice or the like measures the states of color development and lightemission generated from a reaction solution obtained by reacting thesample with a reagent. The sample, the reagent, the reaction solution,and the like used for analysis are contained in a container such as atest tube, and are collected from the container by a collection unitsuch as a dispensing probe.

In the case where bubbles are present on a liquid surface when thesample or reagent is collected, the dispensing probe erroneously detectsthe bubbles as the liquid surface, so that there is a problem that theamount of sample or reagent required for analysis cannot be sucked. Inorder to solve this problem, the technique described in PatentLiterature 1 has been known.

Patent Literature 1 describes “a device that determines the state of asample to be analyzed contained in a container acquires an image of thesample, analyzes the position and size of a target to be detectedrelative to a detection range set in the image by using the image of thesample, and determines the state of the sample on the basis of on theresult of the analysis.”

In recent years, products equipped with a classifier generated byexecuting machine learning are beginning to appear on the market. Theclassifier is generated by machine learning using a set of learning data(learning data set) configured using input data (input signal) andteacher data (teacher signal) input to the classifier.

A classifier (model) such as a neural network is complicated instructure, and it is difficult, for humans to understand the behavior.Therefore, in order to respond to erroneous classification occurring inthe real field, new containers and samples, or customization and tuningfor each facility, a method of constructing a new learning data set,executing machine learning again, and regenerating a classifier isgeneral. This is referred to as re-learning in the specification.

By performing the re-learning, it is expected to improve theclassification accuracy for input data, which has not been able to copewith so far. However, since the characteristics of the classifier arechanged as a result of performing the re-learning, there is apossibility of outputting an erroneous classification result to theinput data correctly recognized before the re-learning. In particular,in the case where the classifier is evaluated using evaluation dataincluding input data and teacher data and strictly defined to be mountedin a product, it is not preferable that the accuracy of classificationfor the evaluation data is deteriorated. It should be noted that theteacher data configuring the evaluation data is also called correctanswer data.

On the other hand, the techniques described in Patent Literature 2 andPatent Literature 3 have been known.

Patent Literature 2 describes “an information processing device 10 formaking an inference using parameters includes: a data acquisition unit31 for acquiring input data; a basic parameter storage unit 41 forstoring a parameter before additional learning; a difference parameterstorage unit 40 for storing a first difference parameter that is adifference between a parameter used for inference and a basic parameter;additional learning means 42 for calculating a difference between aparameter after the additional learning for the basic parameter and thebasic parameter as a second difference parameter; update means 43 forupdating the first difference parameter stored in the differenceparameter storage unit 40 on the basis of the first difference parameterand the second difference parameter; and an inference unit 34 for makingan inference for the input data by using a model parameter generated onthe basis of the basic parameter and the difference parameter updated bythe update means.”

In addition, Patent Literature 3 describes “a parameter estimationdevice that estimates an estimation target parameter value by a neuralnetwork is configured in such a manner that the neural network is set tohave learned by changing a coupling method as previously defined foreach region represented by some of a plurality of input parameters, oneof the regions is determined from some of the plurality of inputparameter values received by region determination means, and thecoupling method is changed by route change means in the same manner aswhen learning the coupling method of the neural network according to theregion determined by the region determination means.”

CITATION LIST Patent Literature

-   [Patent Literature 1] Japanese Unexamined Patent Application    Publication No. 2019-027927-   [Patent Literature 2.1] Japanese Unexamined Patent Application    Publication No. 2017-138808-   [Patent Literature 3] Japanese Unexamined Patent Application    Publication No. Hei 11-85719

SUMMARY OF INVENTION Technical Problem

However, in the technique described in Patent Literature 2, it ispossible to reproduce the characteristics of the classifier at the timeof shipment, but in the case where the classifier actually performs theclassification, a change in the classification result is unavoidablebecause the parameter obtained by adding the basic parameter and thedifference parameter is used.

In addition, the technique described in Patent Literature 3 can beapplied to a case where the number of dimensions of the input signal issmall, but in the case of a multidimensional signal such as an image, itis necessary to generate an enormous amount of classifiers, which isunrealistic. In addition, in machine learning, more learning data isrequired in order to acquire general-purpose feature amounts andclassification performance, and it is not preferable to divide thelearning data into regions.

The present invention proposes a generation method of a classifier thatefficiently and effectively improves the classification accuracy of aclassifier.

Solution to Problem

The following is a representative example of the invention disclosed inthe application. That is, provided is a computing machine having anarithmetic device, a storage device connected to the arithmetic device,and an interface connected to the arithmetic device and generating aclassifier for classifying an arbitrary event, the machine including: astorage unit that stores learning data configured using first input dataand first teacher data; a learning unit that executes learningprocessing for generating the classifier by using the learning datastored in the storage unit; and a generation unit that generates thelearning data, wherein the generation unit calculates a feature amountvector handled by the classifier by using the first input data of thelearning data stored in the storage unit, analyses the distribution ofthe learning data in a feature amount space formed by the feature amountvector on the basis of the feature amount vector of the learning data tospecify a boundary where the classification result of the classifierchanges in the feature amount space, generates first pseudo input databy using the feature amount vector of representative learning data thatis the learning data existing in the vicinity of the boundary, generatesnew learning data configured using the first pseudo input data and thefirst teacher data of the representative learning data, and stores thenew learning data in the storage unit.

Advantageous Effects of Invention

According to the present invention, it is possible to generate aclassifier with the classification accuracy efficiently and effectivelyimproved. Problems, configurations, and effects other than thosedescribed above will become apparent from the following description ofembodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explaining an example of a configuration of anautomatic analysis system of a first embodiment.

FIG. 2 is a diagram for explaining an example of functional blocks of amachine learning device of the first embodiment.

FIG. 3 is a diagram for showing an example of the structure of aclassifier generated by the machine learning device of the firstembodiment.

FIG. 4A is a diagram for explaining changes in the classification resultof a classification unit before and after re-learning of the firstembodiment.

FIG. 4B is a diagram for explaining changes in the classification resultof the classification unit before and after re-learning of the firstembodiment.

FIG. 5 is a flowchart for explaining an example of processing executedby a pseudo sample generation unit of the first embodiment.

FIG. 6A is a diagram for showing an example of the distribution anddistribution density of learning data in a feature amount spacecalculated by the pseudo sample generation unit of the first embodiment.

FIG. 68 is a diagram for showing an example of the distribution anddistribution density of learning data in the feature amount spacecalculated by the pseudo sample generation unit of the first embodiment.

FIG. 7 is a diagram for shoving an example of a method of calculating avector to be generated by the pseudo sample generation unit of the firstembodiment.

FIG. 8 is a flowchart for explaining an example of processing executedby a pseudo sample generation unit of a second embodiment.

FIG. 9 is a diagram for shoving an example of a vector to be generatedcalculated by the pseudo sample generation unit of the secondembodiment.

FIG. 10 is a flowchart for explaining an example of processing executedby a pseudo sample generation unit of a third embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described byusing the accompanying drawings. It should be noted that in thefollowing description and the accompanying drawings, constitutionalelements having the same functions are denoted by the same referencenumerals and duplicate descriptions are omitted. It should be noted thatthe expressions such as “first”, “second”, and “third” in thespecification and the like are added to classify the constitutionalelements and do not necessarily limit a number or an order. It should benoted that the positions, sizes, shapes, ranges, and the like of therespective configurations shown in the drawings and the like do notrepresent the actual positions, sizes, shapes, ranges, and the like insome cases in order to facilitate understanding of the invention.Therefore, the present invention is not limited to the positions, sizes,shapes, ranges, and the like disclosed in the drawings and the like.

First Embodiment

FIG. 1 is a diagram for explaining an example of a configuration of anautomatic analysis system 100 of a first embodiment. The automaticanalysis system 100 includes a machine learning device 101 and anautomatic analysis device 102. A user interface 131 operated by a useris connected to the automatic analysis system 100.

The user interface 131 is configured using input devices such as a mouseand a keyboard and output devices such as a display and a printer. Theinput devices and the output devices may be separate devices.

First, a configuration of the automatic analysis device 102 will bedescribed.

The automatic analysis device 102 includes an imaging unit 124, aclassification unit 12 b, a control unit 121, a collection unit 122, andan analysis unit 123. The hardware is connected to each other via a bus(not shown).

The imaging unit 124 is a device for imaging a container 132 and acollection target 133 contained in the container 132, and is, forexample, a camera. The container 132 is a test tube or the like, and thecollection target 133 is a sample such as blood and urine, a reagent tobe reacted with the sample, a reaction solution obtained by reacting thesample and the reagent, or the like. The imaging unit 124 is installedon the opening side of the container 132, that is, above the container132, and images the container 132 and the collection target 133 fromabove the container 132. An image imaged from above the container 132 bythe imaging unit 124 is referred to as an upper image.

The upper image may be a still image such as BMP, PNG, or JPEG, or aframe image extracted from a moving image such as MPEG or H.264 atregular intervals.

The classification unit 125 classifies the states of the sample and thereagent by inputting the upper image as input data to a classifier. Forexample, the classification unit 125 classifies the presence or absenceof air bubbles and foreign objects on the sample surface that interferewith sample collection. The classifier is generated using machinelearning such as Neural Network and SVM (Support Vector Machine).

The control unit 121 is a device for controlling an operation of thecollection unit 322 on the basis of the classification result outputfrom the classification unit 125, and is, for example, a CPU (CentralProcessing Unit) or the like.

The collection unit 122 is a device for collecting the collection target133 from the container 132, and is, for example, a dispensing probe orthe like.

The analysis unit 123 is a device for performing analysis using thecollection target 133, and is, for example, an immunoanalysis device orthe like. The result of the analysis by the analysis unit 123 isdisplayed on a display (not shown) or the like.

Next, a configuration of the machine learning device 101 will bedescribed.

The machine learning device 101 includes an interface unit 111, anarithmetic unit 112, a memory 113, and a bus 114. The interface unit111, the arithmetic unit 112, and the memory 113 transmit and receiveinformation to and from each other via the bus 114.

The interface unit 111 is a device for connecting to an external device.The machine learning device 101 connects the automatic analysis device102 and the user interface 131 to each other via the interface unit ill.The interface unit 111 receives the upper image from the imaging unit124 and receives a signal input by an operator operating the userinterface 131.

The arithmetic unit 112 is a device for executing various types ofprocessing of the machine learning device 101, and includes, forexample, a CPU, an FPGA (Field-Programmable Gate Array), and the like.The function executed by the arithmetic unit 112 will be described byusing FIG. 2 .

The memory 113 is a device for storing a program executed by thearithmetic unit 112, various information (parameters and coefficients)used by the program, processing results, and the like, and includes anHDD (Hard Disk Drive), a SAM (Random Access Memory), a ROM (Read OnlyMemory), a flash memory, and the like. In addition, the memory 113includes a work area used by the program.

It should be noted that the machine learning device 101 may be providedin a system different from the automatic analysis system 100. In thiscase, the machine learning device 101 transmits and receives informationsuch as images and arithmetic results to and from the automatic analysisdevice 102 via communication means such as a network.

It should be noted that although the interface unit ill is directlyconnected to the imaging unit 124 and the classification unit 125 of theautomatic analysis device 102 in FIG. 1 , the automatic analysis device102 may be provided with a memory and images and processing results ofthe arithmetic unit 112 may be transmitted and received via the memory.

In addition, although the automatic analysis system 100 including theautomatic analysis device 102 will be described as an example of amethod of using the machine learning device 101 in the specification, itis possible to use the machine learning device 101 without limiting tothe automatic analysis device 102 as long as the automatic analysissystem 100 is a system for performing image recognition, imagedetection, segmentation, or the like using machine learning or the like.

FIG. 2 is a diagram for explaining an example of functional blocks ofthe machine learning device 101 of the first embodiment. FIG. 3 is adiagram for showing an example of the structure of a classifiergenerated by the machine learning device 101 of the first embodiment.

The machine learning device 101 includes an input unit 201, a storageunit 202, a learning unit 203, a pseudo sample generation unit 204, aclassifier evaluation unit 205, and an output unit 206. In the firstembodiment, it is assumed that the input unit 201, the storage unit 202,the learning unit 203, the pseudo sample generation unit 204, theclassifier evaluation unit 205, and the output unit 206 are realized bythe arithmetic unit 112 for executing a program. It should be noted thateach of the above-described functional units may be realized usingdedicated hardware.

The input unit 201 receives the upper image from the automatic analysisdevice 102 and stores the same in the storage unit 202. In addition, theinput unit 201 outputs the received upper image to the user interface131 via the output unit 206, and receives a teacher signal for the upperimage from the user. Data obtained by associating the teacher signalwith the upper image is stored as learning data.

The acquisition processing of the teacher signal may be performed in thecase where the upper image is received, may be collectively performedfor a plurality or upper images after a certain period of time elapsessince the upper image is stored in the storage unit 202, or may beperformed in the case where a user request is received.

The output unit 206 outputs the upper image, a learning result, and thelike.

The storage unit 202 stores various information. Specifically, thestorage unit 202 stores information (parameters and coefficient groups)of the classifier used by the classification unit 125 of the automaticanalysis device 102, a learning data set used in the case where theclassifier is generated by machine learning, an evaluation data set usedto evaluate the classifier, an evaluation result of the classifier usingthe evaluation data, and a pseudo sample generated by the pseudo samplegeneration unit 204. It should be noted that a part of the learning datais used as the evaluation data in some cases. In addition, the storageunit 202 stores the upper image received by the input unit 201, andstores the teacher signal input for the upper image by associating withthe upper image. It should be noted that the evaluation result of theclassifier is stored for each generation of learning processing.

Here, the evaluation result of the classifier is a classification suchas the presence or absence or probability of bubbles for the evaluationdata, a classification accuracy for the evaluation data, or the like.

The learning unit 203 will be described.

The learning unit 203 generates information defining the classifier usedby the classification unit 125 of the automatic analysis device 102, forexample, a coefficient group of a neural network by executing machinelearning. In the embodiment, machine learning for generating a neuralnetwork including three fully connected layers as a classifier will bedescribed as an example.

A network 300 shown in FIG. 3 shows a neural network configured usingthree fully connected layers of an input layer 301, a hidden layer 302,and an output layer 303. Each layer includes one or more units.

The input layer 301 is a layer for receiving an input signal, and forexample, the luminance of each pixel of the upper image is input to eachunit.

The hidden layer 302 is a layer for obtaining feature amounts from theinput signal. For example, the unit of the hidden layer 302 substitutesan input signal Xi, a weight Wji, and a bias bj received from each unitof the input layer 301 into Equation (1) to calculate a feature amountYj.

$\begin{matrix}\lbrack {{Equation}1} \rbrack &  \\{{Yj} = {f( {{\sum\limits_{i = 0}^{N - 1}{{Xi} \star {Wji}}} + {bj}} )}} & (1)\end{matrix}$

Here, N represents the number of pixels of the upper image. The functionf is an activation function, such as a Sigmoid function, a tanhfunction, or a ReLU function.

The output layer 303 is a layer for obtaining a final output. Forexample, the feature amount Yj, the weight Wkj, and the bias bk of thehidden layer 302 are substituted into Equation (2) to calculate a finaloutput Zk.

$\begin{matrix}\lbrack {{Equation}2} \rbrack &  \\{{Zk} = {{softmax}( {{\sum\limits_{j = 0}^{H - 1}{{Yj} \star {Wkj}}} + {bk}} )}} & (2)\end{matrix}$

Here, H represents the number of units of the hidden layer 302. Inaddition, softmax is a function for calculating a probability, and isdefined as Equation (3).

$\begin{matrix}\lbrack {{Equation}3} \rbrack &  \\{{{softmax}({Vk})} = \frac{\exp({Vk})}{\sum_{l = 0}^{M - 1}{\exp({Vl})}}} & (3)\end{matrix}$

Here, M represents the number of classes of the classification result.

In the neural network, a loss function representing the differencebetween the final output Zk and a teacher signal T is generallycalculated, and the weight Wji and the weight Wkj for minimizing theloss function are obtained on the basis of the gradient descent method.Negative Log Likelihood, Hinge Loss, or the like is used as the lossfunction.

In the specification, processing of updating a coefficient group such asthe weight Wji and the weight Wkj on the basis of the gradient descentmethod is referred to as learning. In addition, a vector configuredusing the feature amount Yj output from each unit of the hidden layer302 is referred to as a feature amount vector.

It is assumed that the coefficient group calculated by the learningdescribed above is stored in the storage unit 202 as information of theclassifier.

It should be noted that although the neural, network including threefully connected layers is used as an example of the network 300, thenumber of hidden layers may be increased, or a convolutional neuralnetwork or the like may be used instead of the fully connected layers.In this case, the feature amount refers to the output signal of thehidden layer immediately before the output layer (final hidden layer).

In addition, the learning unit 203 of the first embodiment executesre-learning using the learning data set preliminarily stored in thestorage unit 202, the upper image newly input from the input unit 201and the teacher signal associated therewith, and the learning datagenerated by the pseudo sample generation unit 204 to be describedlater. Accordingly, it is possible to obtain a coefficient group havinga higher classification accuracy than when the coefficient group(classifier) stored in the storage unit 202 is used before executing there-learning.

In machine learning, as a method of improving the robustness of theclassifier, the learning data is added by increasing the number ofpatterns of the learning data. The addition of the learning data refersto generation of the learning data on the basis of a method of aperturbation such as the position, rotation angle, and luminance of thelearning data, and generation of the learning data using a generator orthe like.

The addition of the learning data has an effect of improving therobustness of the classifier. However, the number of patterns of thecombinations of perturbations such as the above-described position,rotation angle, and luminance is enormous, and it is not practical touse the learning data of all the combinations for learning. On the otherhand, it is difficult to judge what kind of learning data is effectivein improving the classification accuracy of the classifier. In addition,there is also the following problem by re-learning using the addedlearning data. FIG. 4A and FIG. 4E are diagrams each explaining changesin the classification result of the classification unit 125 before andafter the re-learning of the first embodiment.

Here, a classifier for classifying the presence or absence of bubbleswill be described as an example. The classifier outputs the probabilityof “with bubbles” and “without bubbles”.

FIG. 4A and FIG. 4B show diagrams in which the learning data and theevaluation data are plotted in a feature amount space 400. In thefeature amount space 400, the learning data and the evaluation data arerepresented as feature amount vectors. In the following description, inthe case where the input data of the learning data and the evaluationdata is not distinguished, the data is referred to as a sample. In FIG.4A and FIG. 4B, the types of teacher signals of the classifier areexpressed as the shapes (circles and squares) of samples. In addition,the learning data is expressed in white, and the evaluation data isexpressed in black.

A classification surface 401 represents the boundary of theclassification result of each class in the feature amount space 400. Thesamples positioned above the classification surface 401 show samplesdetermined as “with bubbles”, and the samples positioned below theclassification surface 401 show samples determined as “without bubbles”.

When viewed from the perspective of the probability of “with bubbles”and “without bubbles” output by the classifier, the samples positionedon the classification surface 401 have the same values of bothprobabilities, the samples positioned above the classification surface401 are superior in the probability of “with bubbles”, and the samplespositioned below the classification surface 401 are superior in theprobability of “without bubbles”.

FIG. 4A shows an example of the feature amount space 400 and theclassification surface 401 before re-learning.

Learning data 402 and learning data 403 are data used for learning ofthe classifier. The learning data 402 represents learning data in whichthe teacher signal is “with bubbles”, and the learning data 403represents learning data in which the teacher signal is “withoutbubbles”.

The learning unit 203 updates the classifier so as to output a correctclassification result to the learning data 402 and 403. As a result ofthe learning using the learning data 402 and 403, the classificationsurface 401 is formed.

Evaluation data 404 and evaluation data 405 are data used for evaluatingthe classifier, and are not referred to at the time of learning. Theclassification accuracy and the classification result of the classifierfor the evaluation data are important for the performance of theclassifier. The evaluation data 404 represents evaluation data in whichthe teacher signal is “with bubbles”, and the evaluation data 405represents evaluation data in which the teacher signal is “withoutbubbles”.

In FIG. 4A, the evaluation data 404 is positioned above theclassification surface 401 and the evaluation data 405 is positionedbelow the classification surface 401, which indicates that theclassifier correctly outputs the classification result.

FIG. 4B shows an example of the feature amount space 400 and theclassification surface 401 after re-learning.

The classification surface 401 is changed from the classificationsurface 401 of FIG. 4A by re-learning. The classifier can correctlyclassify the learning data 402 and the learning data 403 as similar tobefore the re-learning. However, the evaluation data 404 indicated bythe circular frame is positioned below the classification surface 401,and the classification result is “without bubbles”. As a result of thechange of the classification surface 401 by re-learning, there is a casewhere erroneous classification that did not occur before the re-learningoccurs.

As reasons that the classification surface 401 changes as describedabove, there are various factors such as the configuration of thelearning data, the order of learning, the configuration of theclassifier, the initial value of the weight of the classifier, theconvergence method of learning, and the number of times of learning.

As one of the methods for making the classification surfaces 401 similarbefore and after the re-learning, there is a method called transferlearning. A random number is used as the initial, value of a weight inordinary learning, but a coefficient group of a classifier beforere-learning is used as the initial value of re-learning in transferlearning. Accordingly, since the classification surface 401 beforere-learning is used as a starting point, the classification surface 401is easily maintained. However, despite that the main purpose ofre-learning is to correct the classification surface for a sample groupthat cannot be correctly classified by the classifier beforere-learning, learning is started from the state in which the samplescannot be correctly classified in the case of transfer learning, andthus it may be more difficult to improve the classification accuracy forerroneous classification than the case where earning is started from arandom number.

In order to solve the above-described problem, the pseudo samplegeneration unit 204 to be described later generates learning data forre-learning on the basis of the distribution of the learning data in thefeature amount space 400.

In the first embodiment, a pseudo sample (input data) for reproducingthe classification surface 401 before re-learning is generated for aregion where the distribution density of the learning data is high inthe feature amount space 400. Accordingly, it is possible to suppress achange in the classification surface 401 in the region where thelearning data is dense, and to improve the classification accuracy byre-learning for a sample that the classifier could not correctlyclassify before re-learning (that is, a region where learning isinsufficient).

As the execution timing of the re-learning, a case in which a certainnumber of sets of new upper images and teacher data are stored in thestorage unit 202, a case in which a certain number of upper images oferroneous classification are stored, and the like are conceivable. Inaddition, in the case where an instruction to start re-learning usingthe user interface 131, a communication device, or the like by a user ora system administrator is received via the input unit 201, re-learningmay be executed.

Next, the pseudo sample generation unit 204 will be described.

FIG. 5 is a flowchart for explaining an example of processing executedby the pseudo sample generation unit 204 of the first embodiment. FIG.6A and FIG. 6B are diagrams each showing an example of the distributionand distribution density of the learning data in a feature amount spacecalculated by the pseudo sample generation unit 204 of the firstembodiment. FIG. 7 is a diagram for showing an example of a method ofcalculating a vector to be generated by the pseudo sample generationunit 204 of the first embodiment.

The pseudo sample generation unit 204 calculates the distributiondensity of the learning data in the feature amount space (Step S501).Here, the feature amount indicates the output of the hidden layer 302 inFIG. 3 , and the feature amount space is an H-dimensional space having Hfeature amounts as axes. Specifically, the following processing isexecuted.

The pseudo sample generation unit 204 inputs learning data (input data)to the network 300 to acquire a feature amount vector of the learningdata and records the same in a work area.

The pseudo sample generation unit 204 calculates the distributiondensity of the learning data by using a feature amount vector group. Asan example of the method of calculating the distribution density, thereis a method of using an H-dimensional kernel for extracting a subspaceof the feature amount space to count the number of data included in thekernel while scanning the feature amount space.

In addition, the pseudo sample generation unit 204 specifies theboundary surface on the basis of the feature amount vector group and theteacher data of the learning data. As a method of specifying theboundary surface, for example, there is a method of obtaining the sameby using the least square method.

It should be noted that in the case where the number of dimensions ofthe feature amount vectors is large, the arithmetic time for specifyingthe boundary surface becomes long, and the amount of computing machineresources to be required also becomes large. Thus, the number ofdimensions of the feature amount vectors may be reduced according to thearithmetic time and the amount of computing machine resources. Forexample, there is a method of reducing the number of dimensions of thefeature amount vectors by executing principal component analysis.

In FIG. 6A and FIG. 6B, it is assumed that H is equal to 2 forsimplifying the explanation. A learning data distribution 601 shown inFIG. 6A shows the distribution of the learning data in the featureamount space using a feature amount Y0 and a feature amount Y1 as axes.A distribution density 602 shown in FIG. 6B is obtained as a result ofobtaining the distribution density of the learning data by setting akernel size of 11×11 and a slide width of 5×5 of the kernel for thelearning data distribution 601. As the density in the kernel is lower,they are displayed to be whiter, and as the density is higher, they aredisplayed to be darker.

The above is the description of the processing of Step S501.

Next, the pseudo sample generation unit 204 extracts representativelearning data positioned in the vicinity of the classification surfaceon the basis of the distribution density of the learning data (StepS502). Specifically, the following processing is executed.

The pseudo sample generation unit 204 specifies a region where thedistribution density of the learning data is high. As a specifyingmethod, for example, there is a method in which a threshold value isprovided in advance and a region where the distribution density is equalto or larger than the threshold value is selected.

The pseudo sample generation unit 204 determines a representative pointfor the specified region. As the representative point, for example, themaximum point, the centroid point, or the like of the distributiondensity in each specified region is used.

The pseudo sample generation unit 204 extracts learning data having aposition closest to the representative point in the feature amountspace. In the following description, the input data of the extractedlearning data is referred to as a neighborhood sample.

A neighborhood sample group is a sample group that largely affects theformation position of the classification surface. In the embodiment, thepseudo samples are generated so as to maintain the positionalrelationship between the neighborhood samples, so that theclassification surface after the re-learning can easily reproduce theclassification surface before the re-learning.

The above is the description of the processing of Step S502.

Next, the pseudo sample generation unit 204 calculates a feature amountvector for generating the pseudo sample (Step S503). In the followingdescription, the feature amount vector is referred to as a vector to begenerated. Specifically, the following processing is executed.

For each neighborhood sample associated with the teacher data of“without bubbles”, the pseudo sample generation unit 204 specifies theneighborhood sample having the shortest distance with each neighborhoodsample associated with the teacher data of “without bubbles” andassociated with the teacher data of “with bubbles”. In the followingdescription, the neighborhood sample associated with the teacher data of“without bubbles” is referred to as a first neighborhood sample, and theneighborhood sample associated with the teacher data of “with bubbles”is referred to as a second neighborhood sample.

The pseudo sample generation unit 204 substitutes a feature amountvector FA of the first neighborhood sample and a feature amount vectorFB of the specified second neighborhood sample into Equation (4) andEquation (5) to calculate a vector to be generated FA′ and a vector tobe generated FB′.

[Equation 4]

FA′=FA+α(FB−FA)  (4)

[Equation 5]

FB′=FB−b(FB−FA)  (5)

Here, (FB-FA) is a vector on the feature amount space representing theposition of the feature amount vector FB with the feature amount vectorFA as the origin. The coefficient a and the coefficient b arecoefficients for determining the magnitude of perturbations for thevector to be generated FA′ and the vector to be generated FB′, and areset in accordance with the distance from the classification surface. Aspecific setting example will be described later.

The relationship between the neighborhood sample and the vector to begenerated will be described in detail by using FIG. 7 .

A neighborhood sample 701 indicates the first neighborhood sample, and aneighborhood sample 702 indicates the second neighborhood sample. Thefeature amount vector of the neighborhood sample 701 is FA, and thefeature amount of the neighborhood sample 702 is FB. The neighborhoodsample 701 is positioned at coordinates separated from theclassification surface 703 only by a distance DA, and the neighborhoodsample 702 is positioned at coordinates separated from theclassification surface 703 only by a distance DB. A vector to begenerated FA′ 704 and a vector to be generated FA″ 706 are vectors to begenerated that are generated from the first neighborhood sample 701, anda vector to be generated FB′705 and a vector to be generated FB″ 707 arevectors to be generated that are generated from the second neighborhoodsample 702.

The vector to be generated FA′ 704 is a vector calculated on the basisof Equation (4), and the vector to be generated FB′705 is a vectorcalculated on the basis of Equation (5). Here, a calculation example ofthe coefficient a is shown in Equation (6).

[Equation 6]

α=DA*r  (6)

According to Equation (6), the coefficient a is a value proportional tothe distance DA. In addition, r is a proportional constant related tothe distance DA and a, and is a real value between 0.0 and 1.0. In FIG.7 , r=0.3. Thus, the vector to be generated FA′ 704 calculated byEquation (4) is positioned at coordinates where the feature amountvector FA is moved only by DA*0.3 from the first neighborhood sample 701toward the second neighborhood sample 702.

Equation (6) describes the coefficient a, but can also be applied to thecoefficient b by replacing the distance DA with the distance DB and thecoefficient a with the coefficient b. Thus, the vector to be generatedFB′705 calculated by Equation (5) is positioned at coordinates where thefeature amount vector FB is moved only by DB*0.3 from the secondneighborhood sample 702 toward the first neighborhood sample 701.

By arranging the vector to be generated in a wide range in proportion tothe distance from the classification surface 703 as described above, itis possible to suppress the classification surface after the re-learningfrom coming excessively close to the neighborhood sample, so that theclassification surface maintaining the positional relationship with theneighborhood sample can be reproduced.

In addition, in order to maintain the center position of thedistribution, the vector to be generated FA″ 706 and the vector to begenerated FB″ 707 may be generated as shown in Equations (7) and (8). Itshould be noted that the positive and negative signs of the coefficientsare reversed between Equations (4) and (5) that are the equations forcalculating the vector to be generated FA′ 704 and the vector to begenerated FB′705.

[Equation 7]

FA″=FA−α(FB−FA)  (7)

[Equation 8]

FB″=FB+b(FB−FA)  (8)

By adding learning data to both sides of the first neighborhood sample701 and the second neighborhood sample 702 that are distributioncenters, changes in the positions of the neighborhood sample 701 and theneighborhood sample 702 on the feature amount space are suppressed.

In addition, although the example of generating a pair of vectors to begenerated has been described above, a plurality of vectors to begenerated may be generated. Equation (9) shows an example of an equationfor calculating the coefficient a in the case where a plurality ofvectors to be generated is generated.

[Equation 9]

α∈[0,α′]  (9)

In the equation, a′ represents the maximum value of the range of thecoefficient a, and, for example, the value calculated by Equation (6) isset to a′, so that a plurality of vectors to be generated can begenerated while randomly changing the coefficient a between 0 and a′. Inaddition, although the example of changing the position of the vector tobe generated according to the distance has been described above, thenumber of vectors to be generated may be changed according to thedistance.

The above is the description of the processing of Step S503.

Next, the pseudo sample generation unit 204 generates an upper image(pseudo sample) that is input data on the basis of the vector to begenerated, and further generates learning data including the upper image(Step S504). The generated learning data is stored in the storage unit202. Specifically, the following processing is executed.

The pseudo sample generation unit 204 copies the neighborhood samplethat is the calculation source of the vector to be generated, and setsthe sample as an additional sample I.

The pseudo sample generation unit 204 updates the additional sample I onthe basis of Equation (10).

$\begin{matrix}\lbrack {{Equation}10} \rbrack &  \\{I_{t + 1} = {I_{t} + \frac{\sigma{❘{Z - F_{t}}❘}}{\sigma I_{t}}}} & (10)\end{matrix}$

Here, It represents time t, and It+1 represents the additional sample Iat time t+1. Z represents the vector to be generated calculated in StepS503, and Ft represents the feature amounts obtained by inputting theadditional sample It to the network 300. By repeatedly executing theoperation of Equation (0.10) and updating It, an image for outputtingthe vector to be generated can be generated.

The pseudo sample generation unit 204 associates the generated image(pseudo sample) with the teacher signal to generate learning data. Itshould be noted that as the teacher signal associated with the generatedimage, the teacher signal of the neighborhood sample that is the copysource of the additional sample I is used.

It should be noted that the above is an example of a generation method,and the pseudo sample may be generated using a generator such as GAN(Generative Adversarial Networks) or VAE (Variational AutoEncoder). Inthe case where the generator is g, an image (input data) can begenerated as similar to the above by an updating formula shown byEquation (11).

$\begin{matrix}\lbrack {{Equation}11} \rbrack &  \\{R_{t + 1} = {R_{t} + \frac{\sigma{❘{Z - {F( {g( R_{t} )} )}}❘}}{\sigma R_{t}}}} & (11)\end{matrix}$

Here, Rt is an input vector to the generator at time t, and thegenerator g uses Rt as an input to generate generation data g (Rt)having the same number of dimensions as the additional sample I. Inaddition, F (g (Rt)) represents a feature amount vector obtained byinputting the generated data g (Rt) into the classifier.

By adding the learning data group generated by the flow described aboveto the learning data set, the positional relationship between theneighborhood samples is maintained, and the classification surfacesimilar to that before the re-learning can be reproduced.

It should be noted that the pseudo sample generation unit 204 generatesthe learning data only for the region where the learning data is dense.Thus, the classification surface can be reproduced only for the regionwhere the learning data is dense, that is, the region where thereliability of the classification result is high. On the other hand, ina region where the learning data is sparse, the classification accuracyis improved by re-learning.

Next, the classifier evaluation unit 205 will be described.

The classifier evaluation unit 205 compares the classification result ofthe classifier before the re-learning with the classification result ofthe classifier after the re-learning, and determines a classifier to beapplied to the classification unit 125 of the automatic analysis device102 on the basis of the comparison result.

Here, an example of an evaluation method will be described. Theclassifier evaluation unit 206 preliminarily stores the evaluationresult for the evaluation data of the classifier before the re-learningin the storage unit 202. The classifier evaluation unit 205 obtains theevaluation result for the evaluation data stored in the storage unit 202by using a classifier newly generated by the learning unit 203. Theclassifier evaluation unit 205 verifies the classification accuracies ofthe classifiers before and after the re-learning, or the difference ofsamples of erroneous classification between the classifiers before andafter the re-learning, and determines a classifier to be applied to theclassification unit 125.

For example, in order for the classifier evaluation unit 205 to verifythat new erroneous classification has not occurred by the re-learning,there is a method in which it is confirmed that a set of samples oferroneous classification in the classifier after the re-learning isincluded in a set of samples of erroneous classification in theclassifier before the re-learning, and the classifier after there-learning is employed in the case where new erroneous classificationhas not occurred. An example of an evaluation formula is shown inEquation (12).

[Equation 12]

|M _(β) −M _(α) |≤;Th _(M)  (12)

Mβ represents an erroneous classification set of the classifier afterthe re-learning, Ma represents an erroneous classification set of theclassifier before the re-learning, | |represents the number of elementsof the set, the minus represents an operation for obtaining a differenceset, and ThM represents an allowable number for the number of erroneousclassifications.

For example, in the case where Equation (12) is satisfied, theclassifier evaluation unit 205 employs the classifier after there-learning, and the classifier evaluation unit 205 employs theclassifier before the re-learning in the case where Equation (12) is notsatisfied. In the case where 0 is set to ThM, it is a condition forupdating the classifier that new erroneous classification does notoccur, and it can be guaranteed that the classification accuracy is notdeteriorated due to the re-learning and updating of the classifier.

In addition to the above, a method of confirming whether or not theclassification accuracy of the classifier after the re-learning is equalto or higher than the classification accuracy of the classifier beforethe re-learning may be used. An evaluation formula is shown in Equation(13).

[Equation 13]

A _(β) −A _(α) ≥Th _(A)  (13)

Aα and Aβ represent classification accuracies for the evaluation databefore and after the re-learning, respectively, and ThA represents athreshold value for the classification accuracy.

In the case where Equation (13) is satisfied, the classifier evaluationunit 205 employs the classifier after the re-learning, and theclassifier evaluation unit 205 employs the classifier before there-learning in the case where Equation (13) is not satisfied.

In addition, the evaluation data may be divided into a plurality ofsets, and the evaluation method may be switched for each set. Forexample, it is possible to employ a method in which evaluation data thatis easily classified or evaluation data having a large influence onanalysis is divided into a first set, evaluation data that, is difficultto classify or evaluation data having a small influence even ifclassification fails is divided into a second set, it is verified that,new erroneous classification has not occurred in the first set and thatthe classification accuracy has not been deteriorated in the second set,and it is verified whether or not both are satisfied. In addition, it ispossible to employ a method in which weights for the classificationaccuracy and the number of erroneous classifications of each set are setin advance, and a comprehensive evaluation is performed on the basis ofthe total value of values obtained by multiplying the classificationaccuracy and the number of erroneous classifications by the weights.

The classifier evaluation unit 205 outputs the evaluation result via theoutput unit 206. For example, the classifier evaluation unit 205outputs, as the evaluation result, information indicating a changebetween the erroneous classifications of the classifiers before andafter the re-learning. In addition, the classifier evaluation unit 205outputs information of the employed classifier as the evaluation result.

According to the first embodiment, the machine learning device 101generates new learning data for reproducing a classification surface fora region where the distribution density of the learning data in thefeature amount space is high. In addition, the machine learning device101 executes re-learning by using a learning data set with new learningdata added, so that it is possible to efficiently improve theclassification accuracy of a region with sparse learning data whilesuppressing variations in the classification result of a region withdense learning data. That is, it is possible to efficiently andeffectively improve the classification accuracy of the classifier.

Second Embodiment

In a second embodiment, in addition to the learning data generated onthe basis of the boundary surface, the machine learning device 101 addsthe learning data to a region where the distribution density of thelearning data in the feature amount space is low.

Since the distribution of the learning data in the feature amount spacedirectly affects the generation of the classification surface, themachine learning device 101 of the second embodiment generates thelearning data in a region where the distribution density of the learningdata in the feature amount space is low. The distribution density oflearning data in an input space formed using the learning data (inputdata) itself tends to be significantly different from the distributiondensity of the learning data in the feature amount space formed usingthe feature amounts handled by the classifier. Therefore, even if thelearning data is added on the basis of the distribution density of thelearning data in the input space, the classification accuracy of theclassifier is not necessarily improved.

Hereinafter, the second embodiment will be described focusing on thedifference from the first embodiment.

The configuration of an automatic analysis system 100 of the secondembodiment is the same as that of the first embodiment. The hardwareconfiguration and the functional block configuration of a machinelearning device 101 of the second embodiment are the same as those ofthe first embodiment. In addition, the hardware configuration and thefunctional, block configuration of an automatic analysis device 102 ofthe second embodiment, are the same as those of the first embodiment.

In the second embodiment, among the functional blocks provided in themachine learning device 101, processing executed by the pseudo samplegeneration unit 204 is partially different. Other functional blocks arethe same as those in the first embodiment.

FIG. 8 is a flowchart for explaining an example of processing executedby the pseudo sample generation unit 204 of the second embodiment. FIG.9 is a diagram for showing an example of a vector to be generatedcalculated by the pseudo sample generation unit 204 of the secondembodiment.

Since the processing from Step S501 to Step S504 is the same as that inthe first embodiment, the explanation thereof will be omitted.

After the processing of Step S504 is completed, the pseudo samplegeneration unit 204 calculates a vector to be generated in a regionwhere the distribution density of the learning data is low (Step S801).Specifically, the following processing is executed.

The pseudo sample generation unit 204 specifies a region where thedistribution density of the learning data is low. As a specifyingmethod, for example, there is a method in which a threshold value isprovided in advance and a region where the distribution density issmaller than the threshold value is selected.

The pseudo sample generation unit 204 randomly selects a plurality ofpoints from the specified region and calculates the selected points as avector to be generated. The method for determining the number of pointsto be selected includes, for example, a method of dividing the specifiedregion into a plurality of grids and selecting so that the number oflearning data included in each grid is equal.

A calculation method of the vector to be generated will be describedwith reference to FIG. 9 . FIG. 9 shows a partial region 901 where Y0 is25 or larger and smaller than 40 and VI is 25 or larger and smaller than50 from learning data distribution 601. The partial region 901 is aregion where distribution density 602 is 0 to 2 and the distributiondensity of the learning data is low. Square marks represent featureamount vectors of the learning data, and X marks represent vectors to begenerated. In FIG. 9 , the vectors to be generated are calculated sothat three pieces of learning data (the feature amount vector of thelearning data and the vector to be generated) are included in a gridobtained by dividing the feature amount space by a constant width. Thatis, more vectors to be generated are calculated in a region where thedensity is lower.

The above is the description of the processing of Step S801.

Next, the pseudo sample generation unit 204 generates an image (pseudosample) on the basis of the vector to be generated, and generateslearning data including the image (Step S802). The generated learningdata is stored in the storage unit 202. Since the processing in StepS802 is similar to that in Step S504, the description thereof will beomitted. At this time, the teacher signal associated with the generatedimage uses the teacher signal of the learning data included in thespecified region. It should be noted that the user may input the teachersignal by referring to the generated image.

By the processing in Step S801 and Step S802, the learning data can beadded mainly to a region where the distribution density of the learningdata in the feature amount space is low.

According to the second embodiment, the machine learning device 101 addslearning data to maintain the boundary surface in a region where thedistribution density of the learning data in the feature amount space ishigh, and adds learning data to a region where the distribution densityof the learning data in the feature amount space is low. In addition,the machine learning device 101 can efficiently improve theclassification accuracy of the classifier by executing re-learning usinga learning data set with new learning data added.

Third Embodiment

In a third embodiment, evaluation data is added to a region where thedistribution density of the evaluation data in the feature amount spaceis low.

In an evaluation of the classifier generated by machine learning,comprehensiveness of the evaluation data is important. However, asdescribed in the second embodiment, even if the evaluation data iscomprehensively distributed in the input space, the distribution of theevaluation data is different in the feature amount space. Therefore,there is a possibility in the feature amount space that a sparse regionand a dense region of the distribution of the evaluation data or aregion where no evaluation data exists is generated. In this case,sufficient robustness cannot be ensured. Thus, there is a possibilitythat erroneous classification occurs after a product (automatic analysisdevice 102) equipped with a classifier is shipped.

Therefore, in the third embodiment, the method described in the secondembodiment is applied to generate evaluation data for improving thequality of the evaluation of the classifier.

Hereinafter, the third embodiment will be described focusing on thedifference between the first embodiment and the second embodiment.

The configuration of an automatic analysis system 100 of the thirdembodiment is the same as that of the first embodiment. The hardwareconfiguration and the functional block configuration of a machinelearning device 101 of the third embodiment are the same as those of thefirst embodiment. In addition, the hardware configuration and thefunctional block configuration of an automatic analysis device 102 ofthe third embodiment are the same as those of the first embodiment.

In the third embodiment, among the functional blocks provider in themachine learning device 101, processing executed by the pseudo samplegeneration unit 204 and the classifier evaluation unit 205 is partiallydifferent. Other functional blocks are the same as those in the firstembodiment.

First, the pseudo sample generation unit 204 will be described. FIG. 10is a flowchart for explaining an example of processing executed by thepseudo sample generation unit 204 of the third embodiment.

The pseudo sample generation unit 204 calculates the distributiondensity of the evaluation data in the feature amount, space (StepS1001). The processing in Step S1001 is processing in which theprocessing target in Step S501 is replaced with the evaluation data.

Next, the pseudo sample generation unit 204 calculates a vector to begenerated in a region where the distribution density of the evaluationdata is low (Step S1002). The processing in Step S1002 is processing inwhich the processing target in Step S801 is replaced with the evaluationdata.

Next, the pseudo sample generation unit 204 generates an image (pseudosample) on the basis of the vector to be generated, and generatesevaluation data including the image (Step S1003). The generatedevaluation data is stored in the storage unit 202. The processing inStep S1003 is processing in which the processing target in Step S802 isreplaced with the evaluation data.

By executing the processing described in FIG. 10 , the evaluation datacan be added mainly to a region where the distribution density of theevaluation data in the feature amount space is low.

Next, the classifier evaluation unit 205 will be described. Theclassifier evaluation unit 205 evaluates the classifier by using theevaluation data preliminarily stored in the storage unit 202 and theevaluation data generated by the pseudo sample generation unit 204. Theevaluation method of the classifier may be a method for evaluating thecomprehensive classification accuracy and erroneous classification ofthe evaluation data, or a method for evaluating the evaluation data as aseparate evaluation data set. The verification method of theclassification accuracy and erroneous classification and the evaluationmethod using a plurality of evaluation data sets are similar to thosedescribed in the first embodiment.

According to the third embodiment, the machine learning device 101 addsevaluation data to a region where the distribution density of theevaluation data in the feature amount space is low. The robustness ofthe classifier can be more accurately evaluated by evaluating theclassifier using the existing evaluation data and the added evaluationdata.

It should be noted that the present invention is not limited to theabove-described embodiments, and includes various modified examples. Inaddition, for example, the embodiments have been described in detail toexplain the configurations in a way that is easy to understand thepresent invention, and the present invention is not necessarily limitedto those including all the configurations described above. In addition,some configurations of each embodiment can be added to, deleted from,and replaced by other configurations.

In addition, some or all of the above-described configurations,functions, processing units, processing means, and the like may berealized by hardware by, for example, designing with an integratedcircuit. In addition, the present invention can be realized by a programcode of software for realizing the functions of the embodiments. In thiscase, a storage medium recording a program code is provided to acomputer, and a processor provided in the computer reads the programcode stored in the storage medium. In this case, the program code itselfread from the storage medium realizes the functions of theabove-described embodiments, and the program code itself and the storagemedium storing the program code configure the present invention. Forexample, a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD(Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, amagnetic tape, a nonvolatile memory card, a ROM, or the like is used asa storage medium for supplying such a program code.

In addition, the program code for realizing the functions described inthe embodiments can be implemented in a wide range of programs orscripting languages such as assembler, C/C++, perl. Shell, PHP, Python,and Java.

Further, the program code of software for realizing the functions of theembodiments may be distributed via a network, and stored in storagemeans such as a hard disk or a memory of a computer or a storage mediumsuch as a CD-RW or a CD-R, and a processor provided in the computer mayread and execute the program code stored in the storage means or thestorage medium.

In the above-described embodiments, the control lines and theinformation lines considered to be necessary in the explanation areshown, but ail the control lines and the information lines in a productare not necessarily shown. All the configurations may be connected toeach other.

1. A computing machine having an arithmetic device, a storage deviceconnected to the arithmetic device, and an interface connected to thearithmetic device and generating a classifier for classifying anarbitrary event, the machine comprising: a storage unit that storeslearning data configured using first input data and first teacher data;a learning unit that executes learning processing for generating theclassifier by using the learning data stored in the storage unit; and ageneration unit that generates the learning data, wherein the generationunit calculates a feature amount vector handled by the classifier byusing the first input data of the learning data stored in the storageunit, analyzes the distribution of the learning data in a feature amountspace formed by the feature amount vector on the basis of the featureamount vector of the learning data to specify a boundary where theclassification result of the classifier changes in the feature amountspace, generates first pseudo input data by using the feature amountvector of representative learning data that is the learning dataexisting in the vicinity of the boundary, generates new learning dataconfigured using the first pseudo input data and the first teacher dataof the representative learning data, and stores the new learning data inthe storage unit.
 2. The computing machine according to claim 1, whereinthe generation unit specifies a region where the learning data is sparseon the basis of the analysis result of the distribution of the learningdata in the feature amount space, generates second pseudo input data byusing the feature amount vector included in the specified region,generates new learning data configured using the second pseudo inputdata and the first teacher data of the learning data included in thespecified region, and stores the new learning data in the storage unit.3. The computing machine according to claim 1, wherein the storage unitstores evaluation data configured using second input data and secondteacher data, wherein the computing machine includes an evaluation unitthat evaluates the classifier by using the evaluation data stored in thestorage unit, and wherein the generation unit calculates the featureamount vector by using the second input data of the evaluation data,analyzes the distribution of the evaluation data in the feature amountspace on the basis of the feature amount vector of the evaluation datato specify a region where the evaluation data is sparse, generatessecond pseudo input data by using the feature amount vector included inthe specified region, generates new evaluation data configured using thesecond pseudo input data and the second teacher data of the evaluationdata included in the specified region, and stores the new evaluationdata in the storage unit.
 4. The computing machine according to claim 1,wherein the first pseudo input data is generated on the basis of thefeature amount vector positioned in the vicinity of the representativelearning data in the feature amount space.
 5. The computing machineaccording to claim 1, wherein the storage unit stores evaluation dataconfigured using second input data and second teacher data, wherein thecomputing machine includes an evaluation unit that evaluates theclassifier by using the evaluation data stored in the storage unit,wherein in the case where the learning data generated by the generationunit is stored in the storage unit after generating a first classifier,the learning unit generates a second classifier by executing thelearning processing again, and wherein the evaluation unit compares anoutput value obtained by inputting the second input data of theevaluation data to the first classifier and the second classifier withthe second teacher data of the evaluation data to acquire classificationresults for the evaluation data of the first classifier and the secondclassifier, stores the classification results in the storage unit,analyzes changes in the classification results for the evaluation dataon the basis of the classification results for the evaluation data ofthe first classifier and the second classifier, and determines which oneof the first classifier and the second classifier is employed on thebasis of the analysis result of changes in the classification resultsfor the evaluation data.
 6. The computing machine according to claim 5,wherein an output unit that generates and outputs display informationfor presenting the analysis result of changes in the classificationresults for the evaluation data is provided.
 7. A learning method of aclassifier executed by a computing machine to classify an arbitraryevent, wherein the computing machine has an arithmetic device, a storagedevice connected to the arithmetic device, and an interface connected tothe arithmetic device, wherein the storage device stores learning dataconfigured using first input data and first teacher data, wherein thelearning method of a classifier comprises: a first step in which thearithmetic device executes learning processing for generating theclassifier by using the learning data stored in the storage device; anda second step in which the arithmetic device generates new learning databy using the learning data stored in the storage device and stores thenew learning data in the storage device, and wherein the second stepincludes; a step in which the arithmetic device calculates a featureamount vector handled by the classifier by using the first input data ofthe learning data stored in the storage device; a step in which thearithmetic device analyzes the distribution of the learning data in afeature amount space formed by the feature amount vector on the basis ofthe feature amount vector of the learning data to specify a boundarywhere the classification result of the classifier changes in the featureamount space; a step in which the arithmetic device generates firstpseudo input data by using the feature amount vector of representativelearning data that is the learning data existing in the vicinity of theboundary; and a step in which the arithmetic device generates the newlearning data configured using the first pseudo input data and the firstteacher data of the representative learning data.
 8. The learning methodof a classifier according to claim 7, wherein the second step includes:a step in which the arithmetic device specifies a region where thelearning data is sparse on the basis of the analysis result of thedistribution of the learning data in the feature amount space: a step inwhich the arithmetic device generates second pseudo input data by usingthe feature amount vector included in the specified region: and a stepin which the arithmetic device generates new learning data configuredusing the second pseudo input data and the first teacher data of thelearning data included in the specified region.
 9. The learning methodof a classifier according to claim 7, wherein the storage device storesevaluation data configured using second input data and second teacherdata, and wherein the learning method of a classifier comprises: a stepin which the arithmetic device calculates the feature amount vector byusing the second input data of the evaluation data; a step in which thearithmetic device analyzes the distribution of the evaluation data inthe feature amount space on the basis of the feature amount vector ofthe evaluation data to specify a region where the evaluation data issparse; a step in which the arithmetic device generates second pseudoinput data by using the feature amount vector included in the specifiedregion; a step in which the arithmetic device generates new evaluationdata configured using the second pseudo input data and the secondteacher data of the evaluation data included in the specified region,and stores the new evaluation data in the storage unit; and a step inwhich the arithmetic device evaluates the classifier by using theevaluation data stored in the storage device.
 10. The learning method ofa classifier according to claim 7, wherein the first pseudo input datais generated on the basis of the feature amount vector positioned in thevicinity of the representative learning data in the feature amountspace.
 11. The learning method of a classifier according to claim 7,wherein the storage device stores evaluation data configured usingsecond input data and second teacher data, and wherein the learningmethod of a classifier comprises: a step in which in the case where thenew learning data is stored in the storage device after generating afirst classifier, the arithmetic device generates a second classifier byexecuting the learning processing again; a step in which the arithmeticdevice compares an output value obtained by inputting the second inputdata of the evaluation data to the first classifier and the secondclassifier with the second teacher data of the evaluation data toacquire classification results for the evaluation data of the firstclassifier and the second classifier; a step in which the arithmeticdevice analyzes changes in the classification results for the evaluationdata on the basis of the classification results for the evaluation dataof the first classifier and the second classifier; and a step in whichthe arithmetic device determines which one of the first classifier andthe second classifier is employed on the basis of the analysis result ofchanges in the classification results for the evaluation data.
 12. Thelearning method of a classifier according to claim 11, wherein a step inwhich the arithmetic device generates and outputs display informationfor presenting the analysis result of changes in the classificationresults for the evaluation data is provided.
 13. An analysis systemcomprises the computing machine according to claim 1, and an analysisdevice that performs analysis using a classifier generated by thecomputing machine.